Imaging apparatus and image processing system

ABSTRACT

An imaging apparatus includes an image sensor, a control unit, a first image processing unit, and a second image processing unit. The image sensor includes a plurality of pixels, each of which includes a photoelectric conversion unit which converts light into an electric charge. The control unit makes a first control line output a signal based on the electric charge of the photoelectric conversion unit to a first signal line, and makes a second control line output a signal based on the electric charge of the photoelectric conversion unit to a second signal line. The first image processing unit performs first image processing for the signal output to the second signal line. The second image processing unit performs second image processing for the signal output to the first signal line based on a result of the first image processing in the first image processing unit.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. application Ser. No.14/673,599, filed Mar. 30, 2015, which claims priority from JapanesePatent Application No. 2014-075746, filed Apr. 1, 2014, which are herebyincorporated by reference herein in their entireties.

BACKGROUND

Field of the Disclosure

The present disclosure relates to an imaging apparatus and an imageprocessing system.

Description of the Related Art

In image pickup using an imaging apparatus, a technique is known inwhich a partial region satisfying specific conditions and a specificobject are detected from image signals, and image processing for theimage signals is controlled based on the detection result. JapanesePatent Application Laid-Open No. 2003-141551 discloses a method fordetecting, from image signals, a human face and the direction of theface. In this technique, the direction of the face is roughly classifiedusing a face detection template for face detection, and then thedirection of the face is calculated in detail based on calculation of aneigenvector using principal component analysis. Japanese PatentApplication Laid-Open No. 2007-13415 discloses a method for extracting aspecular reflection component from image signals based on a dichroicreflection model and estimating a color temperature of a light source ina photographing environment. In this technique, the specular reflectioncomponent is extracted using a difference in a pixel value.

According to the technique of Japanese Patent Application Laid-Open No.2003-141551 described above, a detection method with a small amount ofcalculation and low accuracy is combined with a detection method with alarge amount of calculation and high accuracy, and as a result,information on the face in the image can be acquired with high accuracy.However, when image processing such as correction processing of a colorand brightness of the image signal is performed based on the detectionresult of the face, in the related art, it is necessary to wait forstart of the image processing until the detection processing of the faceis completed, generating a time lag in the processing. Particularly,when the detection method in two steps, such as in Japanese PatentApplication Laid-Open No. 2003-141551 described above, is used, there isa problem of long processing time.

Also in Japanese Patent Application Laid-Open No. 2007-13415, in orderto perform white balance correction for an image signal, it is necessaryto wait for completion of the processing of extracting the specularreflection component from the image signal, and a time lag in theprocessing may be generated. In addition, similarly to the detectionprocessing of the face, when the specular reflection component isextracted in combination of rough and high speed processing andprocessing having high accuracy and a large processing amount, a timelag in the processing may be large.

SUMMARY

An imaging apparatus according to an embodiment of the present inventionincludes an image sensor, a control unit, a first image processing unit,and a second image processing unit. The image sensor includes aplurality of pixels. Each of the plurality of pixels has a photoelectricconversion unit which converts light into an electric charge. Thecontrol unit includes a first control line to output a signal based onthe electric charge of the photoelectric conversion unit to a firstsignal line. The control unit includes a second control line to output asignal based on the electric charge of the photoelectric conversion unitto a second signal line. The first image processing unit is configuredto perform first image processing for the signal output to the secondsignal line. The second image processing unit performs second imageprocessing for the signal output to the first signal line based on aresult of the first image processing by the first image processing unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a structure of an imagingapparatus according to a first exemplary embodiment.

FIG. 2 is a block diagram illustrating a structure of a stacked layersensor according to the first exemplary embodiment.

FIG. 3 is a diagram illustrating a structure of a pixel unit of thestacked layer sensor according to the first exemplary embodiment.

FIG. 4 is a diagram illustrating the flow of image processing accordingto the first exemplary embodiment.

FIGS. 5A to 5C are diagrams illustrating a method for extracting aspecular reflection component according to the first exemplaryembodiment.

FIG. 6 is a block diagram illustrating a structure of an imagingapparatus according to a second exemplary embodiment.

FIG. 7 is a diagram illustrating the flow of image processing accordingto the second exemplary embodiment.

FIG. 8 is a diagram illustrating the flow of the image processingaccording to the second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS First Exemplary Embodiment

Hereinafter, an imaging apparatus according to a first exemplaryembodiment of the present invention will be described with reference toFIGS. 1 to 5C. As the first exemplary embodiment, for example, animaging apparatus which estimates a light source based on a result ofextracting a specular reflection component from image signals, andperforms white balance correction will be described.

FIG. 1 is a block diagram illustrating an exemplary structure of theimaging apparatus according to the first exemplary embodiment of thepresent invention. An imaging apparatus 100 includes an imaging opticalsystem 101, a stacked layer sensor 110, an image processing logic unit120, a display unit 132, an operation unit 133, and an external memory134. The stacked layer sensor 110 includes an image sensor 111 whichconverts an optical image into an image signal, and an image processingunit described later in the same package. The image processing logicunit 120 performs image processing for an image signal output from thestacked layer sensor 110. The recording medium 131 records image data.The display unit 132 presents an image to a user. The operation unit 133receives an operation from the user. The imaging apparatus of FIG. 1also includes a picked-up image signal S101, specular reflectioncandidate region information S102, camera signal processing setting dataS103, a recorded image signal S104, a display image signal S105, a userinstruction input signal S106, and external memory R/W information S107.

The stacked layer sensor 110 includes an image sensor 111, a picked-upimage outputting unit 112, a specular reflection candidate regionextracting unit (first image processing unit) 113, a specular reflectioncandidate region information outputting unit 114, and a control signalinputting unit 115. The image sensor 111 converts an optical image intoan image signal. The specular reflection candidate region extractingunit 113 extracts a region which may contain a specular reflectioncomponent from image signals. The stacked layer sensor 110 also includesa sensor read signal a S111, a sensor read signal b S112, specularreflection candidate region information S113, a sensor driving settingsignal S114, and a specular reflection extraction setting signal S115.

The image processing logic unit 120 includes a picked-up image inputtingunit 121, a development processing unit (second image processing unit)122, a specular reflection candidate region information inputting unit123, a specular reflection extracting unit 124, a control signaloutputting unit 125, and a system control unit 126. The image processinglogic unit 120 also includes a development processing input image signalS121, specular reflection candidate region information S122, camerasignal processing setting information S123, and white balance gaininformation S124.

FIG. 2 is a block diagram illustrating an exemplary structure of thestacked layer sensor 110. The stacked layer sensor 110 includes a rowscanning circuit (control unit) a 202 a and a row scanning circuit(control unit) b 202 b, for selecting an image signal to be output tothe picked-up image outputting unit 112 and the specular reflectioncandidate region extracting unit 113, respectively. The stacked layersensor 110 also includes column analog digital converters (column ADCs)203 a and 203 b which convert an analog signal into a digital signal,and a column operation circuit 204 which scans each column.

The image sensor 111 includes a plurality of pixels 211 arranged in atwo dimensional matrix form. The image sensor 111 includes a transfersignal line a 212 a, a reset signal line a 213 a, a row selection signalline a 214 a. The row selection signal line a 214 a reads an imagesignal from the pixel 211 in accordance with control by the row scanningcircuit 202 a. The read image signal is output to a column signal line215 a as described later. The image sensor 111 also includes a transfersignal line b 212 b, a reset signal line b 213 b, and a row selectionsignal line b 214 b. The row selection signal line b 214 b reads animage signal from the pixel 211 in accordance with control by the rowscanning circuit 202 b. The read image signal is output to a columnsignal line 215 b as described later.

A column signal line a 215 a outputs the sensor read signal Sill to thepicked-up image outputting unit 112 via the column ADC 203 a. A columnsignal line b 215 b outputs the sensor read signal S112 to the specularreflection candidate region extracting unit 113 via the column ADC 203b. As illustrated in FIG. 2, the transfer signal line 212 a, the resetsignal line 213 a, the row selection signal line 214 a, and the columnsignal line 215 a are connected to all the pixels 211. Meanwhile, thetransfer signal line 212 b, the reset signal line 213 b, the rowselection signal line 214 b, and the column signal line 215 b areconnected to some of the pixels 211. That is, the sensor read signalS112 output to the specular reflection candidate region extracting unit113 has thinned pixels with respect to the sensor read signal Silloutput to the picked-up image outputting unit 112. For example, thesensor read signal Sill has horizontal 4096 pixels and vertical 2160pixels. The sensor read signal b (S112) has pixels thinned intohorizontal 400 pixels and vertical 216 pixels.

The row scanning circuits 202 a and 202 b, the image sensor 111, thecolumn ADCs 203 a and 203 b, the specular reflection candidate regionextracting unit 113, the column operation circuit 204, the specularreflection candidate region information outputting unit 114, and thepicked-up image outputting unit 112 are formed in the same semiconductorchip. The image sensor 111 and the specular reflection candidate regionextracting unit 113 are stacked on each other to be disposed. That is,the image sensor 111 is disposed on a light entrance side, and thespecular reflection candidate region extracting unit 113 is disposed onthe back side thereof. The image sensor 111 and the specular reflectioncandidate region extracting unit 113 are electrically connected to eachother. As a result, a speed at which the image sensor 111 outputs animage signal to the specular reflection candidate region extracting unit113 can be higher than a speed at which the image sensor 111 outputs animage signal to the development processing unit 122.

FIG. 3 is a circuit diagram illustrating an exemplary structure of thepixel 211. The pixel 211 includes a photodiode 301, a transfertransistor 302, a floating diffusion (FD) 303, a reset transistor 304,an amplifying transistor 305, and selecting transistors 306 a and 306 b.The photodiode 301 is a photoelectric conversion unit which convertslight into an electric charge, in which the anode is grounded and thecathode is connected to the source of the transfer transistor 302. Thetransfer signal lines 212 a and 212 b are connected to the gate of thetransfer transistor 302. As a result, when the transfer transistor 302is turned on by the transfer signal line 212 a or 212 b, an electriccharge accumulated in the photodiode 301 is transferred to the FD303 tobe accumulated in the FD303. In the amplifying transistor 305, a powersource voltage Vdd is applied to the drain, and the gate is connected tothe FD303. The amplifying transistor 305 amplifies the electric chargeaccumulated in the FD303 to convert the electric charge into a voltagesignal.

The selecting transistor 306 a is disposed between the source of theamplifying transistor 305 and the column signal line 215 a. The gate ofthe selecting transistor 306 a is connected to the row selection signalline 214 a. As a result, when the selecting transistor 306 a is turnedon by the row selection signal line 214 a, a voltage signalcorresponding to a voltage of the FD303 is output to the column signalline 215 a.

Meanwhile, the selecting transistor 306 b is disposed between the sourceof the amplifying transistor 305 and the column signal line 215 b. Thegate of the selecting transistor 306 b is connected to the row selectionsignal line 214 b. As a result, when the selecting transistor 306 b isturned on by the row selection signal line 214 b, a voltage signalcorresponding to the voltage of the FD303 is output to the column signalline 215 b.

In the reset transistor 304, the gate is connected to the reset signallines 213 a and 213 b, the drain is connected to the node of the powersource voltage Vdd, and the source is connected to the FD303. As aresult, when the reset transistor 304 is turned on by the reset signalline 213 a or 213 b, the voltage of the FD303 is reset to the powersource voltage Vdd. The sensor read signal Sill is read by scanningcontrol of the row scanning circuit 202 a. The sensor read signal S112is read by scanning control of the row scanning circuit 202 b.

With reference to FIG. 4, read of an image signal in the imagingapparatus 100 and a flow of image processing for the read image signalwill be described. FIG. 4 illustrates operation timing of the imagingapparatus 100 when a moving image signal is read at a driving period of1/30 second for image processing.

The sensor read signal a (S111) read in the image sensor 111 is outputto the outside of the stacked layer sensor 110 as the picked-up imagesignal S101 by the picked-up image outputting unit 112. The picked-upimage signal S101 is received by the picked-up image inputting unit 121to be supplied to the development processing unit 122 as the developmentprocessing input image signal S121.

As illustrated in FIG. 4, read of a first frame image signal (frame 1)of the sensor read signal a (S111) is started at time t10 and terminatedat time t20. Then, a subsequent frame image signal (frame 2) is readbetween the time t20 and t30. Thereafter, a frame image signal is readperiodically.

The development processing unit 122 is a second image processing unit toperform various correction processing (second image processing) such asgamma correction and color correction to the development processinginput image signal S121. The development processing unit 122 performswhite balance correction (second image processing) in accordance withwhite balance gain calculated by a method described later. Imageprocessing in the development processing unit 122 is performed viaexternal memory R/W information S107 with the external memory 134between the time t30 and t31 in FIG. 4.

That is, an image signal corresponding to the first frame of the movingimage read between the time t10 and t20 is output from the developmentprocessing unit 122 at the time t31 delayed by two frames to be used forrecording image data in the recording medium 131 or displaying an imagein the display unit 132.

Meanwhile, the sensor read signal b (S112) is output to the specularreflection candidate region extracting unit 113 between the time t10 andt11. As illustrated in FIG. 4, read of the sensor read signal b (S112)is completed in shorter time than read of the sensor read signal a(S111). This is because the sensor read signal b (S112) has the thinnednumber of pixels, and the image sensor 111 and the specular reflectioncandidate region extracting unit 113 are connected to each other by thestacked layer structure.

Next, the specular reflection candidate region extracting unit 113 is afirst image processing unit to perform first image processing ofextracting, from the sensor read signal b (S112), a specular reflectioncandidate region in which specular reflection may occur on the surfaceof an object. Specific methods therefor will be described with referenceto FIGS. 5A to 5C. FIGS. 5A and 5B illustrate an image signal dividedinto appropriately sized blocks. In FIGS. 5A and 5B, the image signal isdivided into 8×8=64 blocks. It is assumed that the hatching region onthe surface of the object indicates a region in which specularreflection occurs. The specular reflection candidate region extractingunit 113 extracts, from the divided blocks, a block which may include aregion in which specular reflection occurs to output positioninformation thereof as the specular reflection candidate regioninformation S113. It is considered that the region in which specularreflection occurs has higher brightness of the image signal than aregion therearound in which diffusion reflection occurs. As a result,the specular reflection candidate region extracting unit 113 calculatesaverage brightness in each block, and searches for and extracts a blockhaving average brightness higher than that of a block therearound. Underthis condition, a plurality of blocks which are determined to have ahigh possibility of occurrence of specular reflection may exist. In thiscase, a plurality of information items is also output as the specularreflection candidate region information S113. In FIG. 5B, the extractedblock is indicated by the oblique lines. Processing of extracting theabove-described specular reflection candidate region is performedbetween the time t11 and t12.

The specular reflection candidate region information S113 is output tothe specular reflection candidate region information inputting unit 123of the image processing logic unit 120 from the specular reflectioncandidate region information outputting unit 114.

Next, processing of the specular reflection extracting unit 124 will bedescribed. By extracting a reflection component corresponding tospecular reflection from the development processing input image signalS121 input via the picked-up image inputting unit 121, the specularreflection extracting unit 124 estimates a color of a light source. Aregion corresponding to the specular reflection candidate regioninformation S122 in the development processing input image signal S121is used to reduce processing time.

The processing will be described with reference to FIGS. 5B and 5C. Thespecular reflection extracting unit 124 divides the image signal into aplurality of blocks similarly to the specular reflection candidateregion extracting unit 113. The number of division and the position ofdivision are set to be the same as those of the specular reflectioncandidate region extracting unit 113. In the present exemplaryembodiment, the specular reflection extracting unit 124 divides theimage signal into 8×8 blocks. The specular reflection extracting unit124 generates brightness histogram for a pixel in a block included inthe specular reflection candidate region information S122 to extract aspecular reflection component. FIG. 5C illustrates an example ofbrightness histogram generated by the specular reflection extractingunit 124. The vertical axis of FIG. 5C indicates a brightness signal Yof each pixel. The horizontal axis of FIG. 5C indicates a frequencyindicating the number of pixels included in a period of each brightnesssignal Y. As described above, it is considered that the region in whichspecular reflection occurs has higher brightness than a region in whichspecular reflection does not occur. Therefore, in the brightnesshistogram in FIG. 5C, it is considered that a pixel included in a periodof a high brightness signal Y contains many specular reflectioncomponents. As a result, in the brightness histogram, by calculating adifference in the pixel value between the pixel included in a periodcorresponding to the high brightness signal and a pixel included in aperiod corresponding to a lower brightness signal, a specular reflectionsignal can be extracted. The specular reflection component extracted atthis time indicates a color of a light source in a photographingenvironment. Therefore, the specular reflection extracting unit 124 cancalculate white balance gain suitable for the photographing environmentbased on the extracted specular reflection component. The specularreflection extracting unit 124 outputs the calculated white balance gainto the development processing unit 122. The above-described processingis performed between the time t20 and t21 in the sequence of FIG. 4.Between the time t21 and t22, the white balance gain information S124 isupdated to the development processing unit 122 by the system controlunit 126. As described above, between the time t30 and t31 in FIG. 4,the development processing unit 122 can perform image processingincluding white balance correction.

As described above, the imaging apparatus 100 of the present exemplaryembodiment includes the stacked layer sensor 110 in which the imagesensor 111 and the image processing unit are integrated. The specularreflection candidate region extracting unit (image processing unit) 113in the stacked layer sensor 110 extracts a region having a highpossibility of occurrence of specular reflection. Then, the specularreflection extracting unit (image processing unit) 124 outside thestacked layer sensor 110 extracts a specular reflection component withhigher accuracy.

According to the present exemplary embodiment, processing from the startto the completion of read of the sensor read signal b in FIG. 4 andsubsequent processing of extracting a region having a high possibilityof occurrence of specular reflection based on a brightness signal foreach block are completed during a period from the start to thecompletion of read of the sensor read signal a. In cases other than thepresent exemplary embodiment, first, a thinned image corresponding tothe sensor read signal b needs to be generated based on the sensor readsignal a, and then, extracting processing with a brightness value foreach block needs to be performed. As a result, it takes much time forprocessing. According to the present exemplary embodiment, on the otherhand, the above-described time can be reduced.

In the present exemplary embodiment, a region having a high possibilityof containing the specular reflection component is detected usingbrightness information for each block, and the specular reflectioncomponent is extracted using brightness histogram for each pixel.However, the methods for detecting and extracting specular reflectionare not limited thereto. For example, from the read image signals, apartial region having a brightness value higher than that of a regiontherearound may be detected as a region having a high possibility ofcontaining the specular reflection component. In addition, bycalculating a difference in the pixel value for each pixel andextracting a difference value corresponding to a representative color ofa light source stored in advance from the calculated difference values,the specular reflection component may be extracted. That is, any methodmay be used as long as processing of detecting a region having a highpossibility of containing the specular reflection component andprocessing of extracting the specular reflection component based on theimage signal are performed.

In the present exemplary embodiment, the case where the stacked layersensor 110 outputs position information of the region having a highpossibility of containing the specular reflection component has beendescribed. However, information output from the stacked layer sensor 110is not limited thereto. For example, the stacked layer sensor 110 mayoutput information indicating whether the region having a highpossibility of containing the specular reflection component exists. Inthis case, when the region having a high possibility of containing thespecular reflection component exists, the development processing unit122 performs white balance processing based on the detection of thespecular reflection. When the region does not exist, the developmentprocessing unit 122 performs white balance processing based on thedetection of an achromatic color region of the image signal. As aresult, when appropriate white balance correction cannot be performed bythe white balance processing based on the detection of the specularreflection, wasteful processing can be avoided, and it is possible toreduce the total processing time.

As a result of outputting from the stacked layer sensor 110, an imagesignal in the region having a high possibility of containing thespecular reflection component may be output. In this case, the specularreflection extracting unit 124 is only required to generate a brightnesshistogram for an image signal in the output partial region to extractthe specular reflection component. As a result, the specular reflectionextracting unit 124 can calculate white balance gain without waiting forcompletion of read of the image signal with high resolution to thedevelopment processing unit 122 to thereby reduce the total processingtime.

That is, as long as the stacked layer sensor 110 outputs informationindicating a result of detecting the region having a high possibility ofcontaining the specular reflection component, information output fromthe stacked layer sensor 110 may have any format. In the presentexemplary embodiment, the case of using the stacked layer sensor 110including the image sensor 111 and the specular reflection candidateregion extracting unit (image processing unit) 113 connected to eachother by the stacked layer structure has been described. However, thestructure of the sensor is not limited to the stacked layer sensor 110.That is, any sensor in which the pixel 211, the row scanning circuits202 a and 202 b, and the specular reflection candidate region extractingunit 113 are included as an integrated type package may be used.

In the present exemplary embodiment, the imaging apparatus 100 includingthe stacked layer sensor 110 and the image processing logic unit 120 hasbeen described. However, the present exemplary embodiment is not limitedto the imaging apparatus. For example, an image processing systemincluding an imaging apparatus provided with the stacked layer sensor110, the specular reflection candidate region extracting unit 113, and arecording unit, and an image processing apparatus provided with a readunit of a recorded signal and the image processing logic unit 120 may beused. In this case, the recording unit of the imaging apparatus onlyneeds to record an image signal output from the image sensor 111 andinformation of an image region having a high possibility of containingthe specular reflection component. In this way, the image processinglogic unit 120 can reduce the total processing time even when the imageprocessing is performed outside the imaging apparatus.

Second Exemplary Embodiment

Next, as a second exemplary embodiment of the present invention, forexample, an imaging apparatus which detects a face from image signals tocontrol image processing based on a result of the detection will bedescribed. FIG. 6 is a block diagram illustrating an exemplary structureof the imaging apparatus according to the second exemplary embodiment ofthe present invention. The same reference signs are given to similarcomponents to those in the first exemplary embodiment illustrated inFIG. 1, and description thereof will be omitted. Hereinafter,differences between the present exemplary embodiment and the firstexemplary embodiment will be described.

The imaging apparatus in FIG. 6 includes face candidate coordinateinformation S601 and face candidate image information S602. The imagingapparatus also includes a face candidate detecting unit 610, a facecandidate normalizing unit 611, a face candidate image memory 612, aface candidate image outputting unit 613, a face candidate coordinatememory 614, a face candidate coordinate outputting unit 615, and a facetemplate image memory 616. The imaging apparatus also includesindividual face candidate image information S610, normalized facecandidate image information S611, face candidate image information S612,face template image information S613, face template setting informationS614, individual face candidate coordinate information S615, and facecandidate coordinate information S616.

An image processing logic unit 120 includes a face candidate imageinputting unit 621, a face determination unit 622, a face candidatecoordinate inputting unit 623, and an eigenvector memory 624. The imageprocessing logic unit 120 also includes face candidate image informationS621, face candidate coordinate information S622, eigenvectorinformation S623, and face determination result information S624.

Since an image sensor 111 has the structure described in the firstexemplary embodiment, detailed description thereof will be omitted. Asin the first exemplary embodiment, a sensor read signal a (S111) is animage signal of all the pixels of the image sensor 111. A sensor readsignal b (S112) is an image signal having thinned pixels with respect toall the pixels of the image sensor 111. For example, the sensor readsignal a (S111) has horizontal 4096 pixels and vertical 2160 pixels. Thesensor read signal b (S112) has pixels thinned into horizontal 400pixels and vertical 216 pixels.

Next, with reference to FIGS. 7 and 8, operation timing of the imagingapparatus according to the second exemplary embodiment of the presentinvention will be described. FIG. 7 illustrates operation timing duringmoving image capturing operation at a period of 1/30 second. FIG. 8illustrates operation timing during moving image capturing operation ata period of 1/60 second.

The sensor read signal a (S111) is output to the outside of the stackedlayer sensor 110 as the picked-up image signal S101 by the picked-upimage outputting unit 112. The picked-up image signal S101 is receivedby the image processing logic unit 120 from the picked-up imageinputting unit 121 to be supplied to the development processing unit 122as the development processing input image signal S121. As illustrated inFIG. 7, read of the sensor read signal a (S111) is started at the timet70, and terminated at the time t80. Then, a moving image is outputperiodically between the time t90 and t100.

Based on the face determination result information S624, the developmentprocessing unit 122 performs image processing including face regionimage processing, such as skin color correction processing for a faceregion of an object and noise suppression processing specialized for theface region of the object. The image processing is performed using theexternal memory 134 via the external memory R/W information S107.Between the time t90 and t91 in FIG. 7, the display image signal S105and the recorded image signal S104 are output from the developmentprocessing unit 122. That is, an image of the first frame of the sensorread signal a (S111) read between the time t70 and t80 is output fromthe development processing unit 122 between the time t90 and t91(delayed by two frames). Pointer information indicating the face regionof the object is superimposed to the display image signal S105 to bedisplayed in the display unit 132. The recorded image signal S104 isrecorded in the recording medium 131.

Meanwhile, the sensor read signal b (S112) is input to the facecandidate detecting unit 610. The sensor read signal b (S112) is thinnedinto horizontal 400 pixels and vertical 216 pixels, and thus read iscompleted between the time t70 and t71 in FIG. 7.

In the face template image memory 616, a face template image is recordedin advance in a plurality of face sizes. A face as an object changes insize variously depending on the condition during photographing.Therefore, the face template image is also preferably recorded in aplurality of sizes. For example, for each of the horizontal pixel andthe vertical pixel, four types of pixels (160 pixels, 80 pixels, 40pixels, and 20 pixels) are recorded. From the face template settinginformation S614, it is possible to designate which face size of theface template image is to be read as the face template image informationS613.

Using the face template image information S613 in a designated facesize, the face candidate detecting unit 610 searches all the fieldangles of the sensor read signal b (S112) for a partial region havinghigh correlation with the face template image information S613 in thedesignated face size. That is, the face candidate detecting unit 610extracts a partial region image in the same size as the face templateimage information S613 in the designated face size from the sensor readsignal b (S112) while shifting the partial region. By comparison of acorrelation coefficient or a difference between the partial region andthe face template image information S613 in the designated face size,the face candidate detecting unit 610 roughly and correlatedly rateswhether a possibility of existence of the face in the partial region ishigh or low. As described above, the face as the object changes in sizevariously depending on the condition during photographing. Therefore,search processing is desirably performed for all the face templateimages in the plurality of sizes recorded in advance.

Therefore, as illustrated in FIG. 7, between the time t71 and t72, theface candidate detecting unit 610 performs search processing to the facetemplate image information S613 having horizontal 160 pixels andvertical 160 pixels. Between the time t72 and t73, the face candidatedetecting unit 610 performs search processing to the face template imageinformation S613 having horizontal 80 pixels and vertical 80 pixels.Between the time t73 and t74, the face candidate detecting unit 610performs search processing to the face template image information S613having horizontal 40 pixels and vertical 40 pixels. Between the time t74and t75, the face candidate detecting unit 610 performs searchprocessing to the face template image information S613 having horizontal20 pixels and vertical 20 pixels.

If it is determined by the above-described search processing that thepossibility of existence of the face is high, the face candidatedetecting unit 610 generates the individual face candidate coordinateinformation S615 including the face position and size information fromthe information of the partial region compared with the face templateimage information S613 in the designated face size. The face candidatedetecting unit 610 stores the individual face candidate coordinateinformation S615 in the face candidate coordinate memory 614. Aplurality of partial regions may be determined to have a highpossibility of existence of the face. Therefore, a plurality of piecesof individual face candidate coordinate information S615 may also exist.

In addition, the face candidate detecting unit 610 cuts out the partialregion image compared with the face template image information S613 inthe designated face size, and generates the individual face candidateimage information S610 to output the information to the face candidatenormalizing unit 611. The individual face candidate image informationS610 has a plurality of image sizes in association with the facetemplate image information S613 in the designated face size. As aresult, the face candidate normalizing unit 611 performs normalizationprocessing to unify the image sizes to a common image size. For example,the face candidate normalizing unit 611 performs reduction processingsuch that each of the horizontal and vertical pixels has 20 pixels insize. In this way, the normalized face candidate image information S611is generated to be stored in the face candidate image memory 612 as aface candidate image having the horizontal 20 pixels and the vertical 20pixels common in size.

Normalization processing of the face candidate normalizing unit 611 isperformed between the time t71 and t72, between the time t72 and t73,between the time t73 and t74, and between the time t74 and t75 in FIG.7. As illustrated in FIG. 7, processing from the start to the completionof read of the sensor read signal b (S112), and subsequent templatematching processing by the face template images in four face sizes arecompleted during a period from the start to the completion of read ofthe sensor read signal a (S111).

Next, the face candidate coordinate memory 614 and the face candidateimage memory 612 output the recorded face candidate coordinateinformation S616 and face candidate image information S612. At the timet80 in FIG. 7, the face candidate coordinate information S616 and theface candidate image information S612 are output to the outside of thestacked layer sensor 110 from the face candidate coordinate outputtingunit 615 and the face candidate image outputting unit 613 as the facecandidate coordinate information S602 and the face candidate imageinformation S601, respectively. The face candidate coordinateinformation S602 and the face candidate image information S601 are inputto the image processing logic unit 120 via the face candidate coordinateinputting unit 623 and the face candidate image inputting unit 621,respectively.

The face candidate coordinate inputting unit 623 outputs the facecandidate coordinate information S602 to the face determination unit 622as the face candidate coordinate information S622. The face candidateimage inputting unit 621 outputs the face candidate image informationS601 to the face determination unit 622 as the face candidate imageinformation S621. As described above, the face candidate imageinformation S621 is determined by rough rating to have a highpossibility of being the face by a correlation coefficient or adifference between the face template image information S613 and thepartial region image in the same size. Therefore, the accuracy rate isnot so high.

In order to raise the accuracy rate, the eigenvector memory 624 recordsin advance the eigenvector information S623 of the face extracted byprincipal component analysis as data indicating a feature amount of aface image with accuracy. The face determination unit 622 finallyperforms face determination to indicate face or not by checking the facecandidate image information S621 with the eigenvector information S623of the face. The eigenvector information S623 of the face is recoded inadvance in the eigenvector memory 624, output as the eigenvectorinformation S623, and input to the face determination unit 622. Asdescribed above, the face determination unit 622 performs facedetermination processing by principal component analysis and finallyperforms face determination to indicate face or not. The facedetermination unit 622 outputs the face determination result informationS624 to the development processing unit 122 as a result of the facedetermination.

Between the time t80 and t81 in FIG. 7, the face determination unit 622performs final face determination processing for the face candidateimage information S621. Between the time t81 and t82 in FIG. 7, settingparameters of various correction processing controlled based on the facedetermination result information S624 in the development processing unit122 are updated. As described above, between the time t90 and t91 inFIG. 7, by updating the setting parameters in the development processingunit 122, the development processing unit 122 performs image processingincluding face region image processing, such as skin color correctionprocessing for a face region of an object and noise suppressionprocessing specialized for the face region of the object.

In the present exemplary embodiment, processing from the start to thecompletion of read of the sensor read signal b (S112), and subsequenttemplate matching processing by the face template images in four facesizes are completed during a period from the start to the completion ofread of the sensor read signal a (S111).

In cases other than the present exemplary embodiment, first, a thinnedimage corresponding to the sensor read signal b (S112) needs to begenerated based on the sensor read signal a (Sill). Furthermore, then,template matching processing by the face template images in four facesizes needs to be performed. As a result, it takes much time. Accordingto the present exemplary embodiment, the above-described time isconsiderably reduced.

Next, a case where a user has changed control parameters for the imagingapparatus 100 of the present exemplary embodiment will be described.Here, it is assumed that the user has input an instruction to set aframe rate during moving image capturing from a period of 1/30 second(FIG. 7) to a period of 1/60 second (FIG. 8) which is a double speedframe rate via the operation unit 133.

The system control unit 126 sets the camera signal processing settinginformation S123 to a period of 1/60 second to output the camera signalprocessing setting information S123 to the outside of the imageprocessing logic unit 120 via the control signal outputting unit 125 asthe camera signal processing setting data S103. The control signalinputting unit 115 of the stacked layer sensor 110 inputs the camerasignal processing setting data S103. The control signal inputting unit115 outputs a face template setting signal S617 to the face candidatedetecting unit 610 and the sensor driving setting signal S114 to theimage sensor 111. The sensor driving setting signal S114 is a settingsignal which changes setting for thinning the sensor read signal b(S112) generated in the image sensor 111. The face template settingsignal S617 is a setting signal which determines a lower limit of theface size to perform template matching processing among theabove-described face template images in four face sizes.

In the present exemplary embodiment, even when the frame rate is set toa period of 1/60 second which is a double speed frame rate, a period ofprocessing from the start to the completion of read of the sensor readsignal b (S112), and subsequent template matching processing by the facetemplate images in four face sizes is reduced. During the period, inorder to complete the above-described processing during the period fromthe start to the completion of the sensor read signal a (S111), thefollowing change is made.

That is, first, by the sensor driving setting signal S114, setting forthinning the sensor read signal b (S112) is performed so that horizontal400 pixels and vertical 216 pixels are further thinned into horizontal320 pixels and vertical 180 pixels. As a result, a period from the timet120 to t121 in FIG. 8, which is the period from the start to thetermination of read of the sensor read signal b (S112), is furtherreduced.

In addition, by setting the face template setting signal S617, the imagesize of the face template used for template matching processing islimited to the face template image having horizontal 160 pixels andvertical 160 pixels, and the face template image having horizontal 80pixels and vertical 80 pixels. Between the time t121 and t122 in FIG. 8,the face candidate detecting unit 610 performs search processing to theface template image having horizontal 160 pixels and vertical 160pixels. Between the time t122 and t123, the face candidate detectingunit 610 performs search processing to the face template image havinghorizontal 80 pixels and vertical 80 pixels to terminate the templatematching processing.

In this way, the template matching processing time of the face candidatedetecting unit 610 is reduced to ½. In addition, the period from thetime t120 to t121 in FIG. 8, which is the period from the start to thetermination of read of the sensor read signal b (S112), is reduced.Processing from the start to the completion of read of the sensor readsignal a (S111) is performed in 1/60 second (double speed). Even in thiscase, processing from the start to the completion of read of the sensorread signal b (S112) and subsequent template matching processing can becompleted during a period from the start to the completion of read ofthe sensor read signal a (S111).

As described above, according to the present exemplary embodiment, forthe region of all the pixels of the image sensor 111, search processingfor a partial region having high correlation with a face template undera predetermined condition can be completed in parallel during the periodof read of all the pixels of the image sensor 111. In addition,immediately after the completion, face determination processing which isdetailed determination processing in the image processing logic unit 120can be performed. As a result, face detection processing in an objectcan be performed more efficiently.

According to the present exemplary embodiment, even when the read timefor all the pixels of the image sensor 111 changes, it is possible toeasily perform control for completing search processing for a partialregion having high correlation with a face template within the time ofread of all the pixels of the image sensor 111.

In the first and second exemplary embodiments, the row scanning circuit(control unit) 202 a makes the transfer signal line (first control line)212 a output a signal based on an electric charge of the photodiode 301to the column signal line (first signal line) 215 a. The row scanningcircuit (control unit) 202 b makes the transfer signal line (secondcontrol line) 212 b output a signal based on an electric charge of thephotodiode 301 to the column signal line (second signal line) 215 b. Thespecular reflection candidate region extracting unit 113 and the facecandidate detecting unit 610 (and the face candidate normalizing unit611) are first image processing units which perform first imageprocessing for the signal output to the column signal line 215 b. Thedevelopment processing unit 122 is a second image processing unit whichperforms second image processing for the signal output to the columnsignal line 215 a based on a result of the first image processing by thespecular reflection candidate region extracting unit 113 or the facecandidate detecting unit 610 (and the face candidate normalizing unit611).

The row scanning circuit 202 a makes the transfer signal line 212 aoutput a signal based on the electric charge of the photodiodes 301 ofall of the plurality of pixels 211 to the development processing unit122. The row scanning circuit 202 b makes the transfer signal line 212 boutput a signal based on the electric charge of the photodiodes 301 of apart of the plurality of pixels 211 to the specular reflection candidateregion extracting unit 113 or the face candidate detecting unit 610. Thespecular reflection candidate region extracting unit 113 or the facecandidate detecting unit 610 (and the face candidate normalizing unit611) performs first image processing for the signal based on theelectric charge of the photodiodes 301 of some of the pixels 211. Thedevelopment processing unit 122 performs second image processing for asignal based on the electric charge of the photodiodes 301 of all of thepixels 211. The number of the pixels 211 output from the row scanningcircuit 202 b to the column signal line 215 b via the transfer signalline 212 b is smaller than that of the pixels 211 output from the rowscanning circuit 202 a to the column signal line 215 a via the transfersignal line 212 a.

The specular reflection candidate region extracting unit 113 or the facecandidate detecting unit 610 (and the face candidate normalizing unit611) completes the first image processing before the developmentprocessing unit 122 inputs all the signals of the pixel 211 output tothe column signal line 215 a.

The first image processing unit such as the specular reflectioncandidate region extracting unit 113 or the face candidate detectingunit 610 (and the face candidate normalizing unit 611) detects apredetermined image region. Specifically, the first image processingunit detects an image region including a specular reflection componenton the surface of an object, an image region including a face of anobject person, or an image region including a specified object. Thefirst image processing unit outputs presence or absence of apredetermined image region, the number of the predetermined imageregion, and position information of the predetermined image region, oroutputs an image signal of the predetermined image region. Thedevelopment processing unit 122 performs second image processing for theimage region detected by the first image processing unit.

The image processing system includes an imaging apparatus and an imageprocessing apparatus. The imaging apparatus includes the image sensor111, the row scanning circuits 202 a and 202 b, a first image processingunit, and a recording unit. The first image processing unit is thespecular reflection candidate region extracting unit 113 or the facecandidate detecting unit 610 (and the face candidate normalizing unit611). The image processing apparatus includes the development processingunit 122. The recording unit records a signal of the column signal line215 a and a result of the first image processing in the first imageprocessing unit. The development processing unit 122 performs secondimage processing for the signal of the column signal line 215 a recodedin the recoding unit based on the result of the first image processingin the first image processing unit recoded in the recoding unit.

According to the first and second exemplary embodiments, the developmentprocessing unit 122 performs second image processing for the signal ofthe column signal line 215 a based on the result of the first imageprocessing in the first image processing unit. Particularly, thedetection processing is performed in two steps. One of the steps isrough and high-speed detection processing by the specular reflectioncandidate region extracting unit 113 or the face candidate detectingunit 610, and the other is accurate detection processing with a largeamount of processing by the specular reflection extracting unit 124 orthe face determination unit 622. The development processing unit 122 canreduce a time lag generated by waiting for completion of the detectionprocessing to reduce the total processing time.

Note that the above-described exemplary embodiments present onlyspecific examples to implement the present invention. The technicalscope of the present invention should not be construed as being limitedby the exemplary embodiments. That is, the present invention can beimplemented in various forms without deviating from the technical ideaor the main characteristics thereof.

Other Exemplary Embodiments

Exemplary embodiments of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions recorded on a storage medium (e.g.,non-transitory computer-readable storage medium) to perform thefunctions of one or more of the above-described exemplary embodiment(s)of the present invention, and by a method performed by the computer ofthe system or apparatus by, for example, reading out and executing thecomputer executable instructions from the storage medium to perform thefunctions of one or more of the above-described exemplary embodiment(s).The computer may comprise one or more of a central processing unit(CPU), micro processing unit (MPU), or other circuitry, and may includea network of separate computers or separate computer processors. Thecomputer executable instructions may be provided to the computer, forexample, from a network or the storage medium. The storage medium mayinclude, for example, one or more of a hard disk, a random-access memory(RAM), a read only memory (ROM), a storage of distributed computingsystems, an optical disk (such as a compact disc (CD), digital versatiledisc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memorycard, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. An image sensor stacked on an image sensor layerand an signal processing layer, comprising: a plurality of pixels whichconverts an optical image into an image signal; a first signal processorwhich detects a predetermined image region based on the image signal; afirst output terminal which outputs the image signal to a second signalprocessor included in an external device; a second output terminal whichoutputs output region information related to the predetermined imageregion for processing in the second signal processor, and an inputterminal which receives setting data for the first signal processor;wherein the image sensor layer includes the plurality of pixels, andwherein the signal processing layer includes at least the first signalprocessor.
 2. The image sensor according to claim 1, wherein the signalprocessing layer includes the first output terminal, the second outputterminal, and the input terminal.
 3. The image sensor according to claim1, further comprising a first signal line, connecting to the pluralityof pixels, for outputting the image signal to the first output terminal,and a second signal line, connecting to the plurality of pixels, foroutputting the image signal to the first signal processor.
 4. The imagesensor according to claim 1, wherein the first signal processorcompletes detecting processing to detect the predetermined image regionbefore the first output terminal outputs all the image signals for thepixels.
 5. The image sensor according to claim 1, wherein the firstsignal processor is configured to detect an image region including aspecular reflection component on a surface of an object, an image regionincluding a face of an object person, or an image region including aspecified object.
 6. The image sensor according to claim 1, wherein thefirst signal processor performs template matching processing for theimage signal.
 7. The image sensor according to claim 1, wherein thefirst signal processor detects the predetermined image region based onthe brightness of the image signal.
 8. The image sensor according toclaim 1, wherein the first signal processor detects a plurality of thepredetermined image regions based on the image signal.
 9. The imagesensor according to claim 1, further comprising a first control unit anda second control unit which is different from the first control unit forselecting an image signal to be output from the plurality of pixels. 10.The image sensor according to claim 3, further comprising a selectingswitch configured to select whether the image signal is to be output tothe first signal line or the second signal line.
 11. The image sensoraccording to claim 1, wherein data amount of the image signal which isprocessed by the first signal processor is less than that of the imagesignal processed by the second signal processor included in an externaldevice.
 12. The image sensor according to claim 1, further comprising afirst memory in which a plurality of setting data is recorded inadvance, wherein the first signal processor acquires a plurality ofparameters sequentially form the first memory to detect thepredetermined region.
 13. The image sensor according to claim 12,further comprising a second memory in which a detection result of thefirst signal processor is recorded.
 14. The image sensor according toclaim 12, wherein an amount of the plurality of parameters acquired fromthe first memory is limited based on a frame rate.